After SAS Security Updates are applied to a SAS^® 9.4M5 (TS1M5) environment, attempts to restore the SAS Content Server by using the sas-recover-offline command in the SAS Deployment Backup and Recovery Tool might fail with an unknown error code.

This problem is due to an updated Java BeanShell file whose references are not correctly modified in SAS Deployment Backup and Recovery Tool logic. The sas-recover-offline command, or other SAS Deployment Backup and Recovery Tool commands, then fails with a generic "999" unknown error code. When this problem occurs, you might see Java stack information appear in the command output that is similar to the following:

java.lang.ClassNotFoundException: org.apache.tools.ant.BuildException
     at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
     at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
     at java.security.AccessController.doPrivileged(Native Method)
     at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
     at com.sas.app.AppClassLoader.findClass(AppClassLoader.java:486)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
     at com.sas.app.AppClassLoader.loadClass(AppClassLoader.java:618)
     at com.sas.app.AppClassLoader.loadClass(AppClassLoader.java:597)
     at java.lang.Class.forName0(Native Method)
     at java.lang.Class.forName(Class.java:278)
     at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:683)
     at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1863)
     at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1746)
     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2035)
     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
     at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2280)
     at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:558)
     at java.lang.Throwable.readObject(Throwable.java:914)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1161)
     at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2171)
     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2062)
     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:428)
     at com.sas.svcs.backup.dao.BackupServerHelperDAO.populateResult(BackupServerHelperDAO.java:934)
     at com.sas.svcs.backup.dao.BackupServerHelperDAO.recover(BackupServerHelperDAO.java:352)
     at com.sas.svcs.backup.spi.ContentServerBackupSupportService.doSourceRecovery
       (ContentServerBackupSupportService.java:298)
     at com.sas.svcs.backup.spi.BackupSupportServiceProxyImpl.beginSourceRecovery
       (BackupSupportServiceProxyImpl.java:408)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
     at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
       (ReflectiveMethodInvocation.java:183)
     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed
       (ReflectiveMethodInvocation.java:150)
     at org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call
       (AsyncExecutionInterceptor.java:95)
     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
     at java.lang.Thread.run(Thread.java:748)

This stack-trace information does not always appear when this problem occurs. Instead, or in addition to that information, you might see a failure reported for the SAS Content Server in the SAS Deployment Backup and Recovery Tool command output:

sourceType:     contentServer
status: failed
serverSideReturnCode:   999

To determine whether this problem is applicable to your environment, examine the backupserver.log file on your primary middle-tier host. This file is in the directory SAS-configuration-directory/LevN/Backup/. When the problem has occurred, you should see messages similar to the following:

The updated Java BeanShell file is delivered as a part of the SAS Security Update 2018-09 package, which can be applied only on SAS 9.4M5. SAS^® 9.4M4 (TS1M4) and any earlier maintenance releases of SAS 9.4 are not affected by this problem, because the Java BeanShell is not updated in those releases, even when all compatible SAS Security Update packages are applied. SAS^® 9.4M6 (TS1M6) and any later SAS 9.4 releases install the updated BeanShell by default. Applying the SAS Security Updates on these later platforms does not require modification to the referenced files. Therefore, this problem occurs only on SAS 9.4M5 systems.

To resolve this issue, follow the steps below:

1. Download the updated cmu_import_config.xml file from the Downloads tab of this SAS Note.

2. On the SAS server machine, navigate to SASHOME/SASDeploymentManager/9.4/products/cfgwizard__94492__prt__xx__sp0__1/Utilities/. 

3. On the SAS server machine, rename the cmu_import_config.xml file that is stored in this directory to cmu_import_config.xml.orig.

4. Transfer the updated cmu_import_config.xml file, which you downloaded in Step 1, to the SAS server machine.

5. Move or copy the updated cmu_import_config.xml file from Step 4 into the directory SASHOME/SASDeploymentManager/9.4/products/cfgwizard__94492__prt__xx__sp0__1/Utilities/. 

6. Verify that the updated file from Step 5 is located at this path:
   
   SASHOME/SASDeploymentManager/9.4/products/cfgwizard__94492__prt__xx__sp0__1/Utilities/cmu_import_config.xml 

7. If the file has a different name, it is not detected by the SAS Deployment Backup and Recovery Tool, and processes can fail with other errors.

Note: If you have a multimachine SAS server environment, repeat Steps 2–6 for all machines (metadata, compute middle-tier, and so on) that contain the same directory path from Step 2. If any machine contains the old version of the file, the issue will continue to occur.

After the cmu_import_config.xml file is updated on all applicable machines, the failing SAS Deployment Backup and Recovery Tool commands should begin working immediately. No service or system restarts are required for the fix to take effect.

Contact SAS Technical Support if you have questions about performing these steps or need help with validating if this issue applies to your environment.

Operating System and Release Information